Data quality considerations for evaluating COVID-19 treatments using real world data: learnings from the National COVID Cohort Collaborative (N3C).

BACKGROUND: Multi-institution electronic health records (EHR) are a rich source of real world data (RWD) for generating real world evidence (RWE) regarding the utilization, benefits and harms of medical interventions. They provide access to clinical data from large pooled patient populations in addition to laboratory measurements unavailable in insurance claims-based data. However, secondary use of these data for research requires specialized knowledge and careful evaluation of data quality and completeness. We discuss data quality assessments undertaken during the conduct of prep-to-research, focusing on the investigation of treatment safety and effectiveness. METHODS: Using the National COVID Cohort Collaborative (N3C) enclave, we defined a patient population using criteria typical in non-interventional inpatient drug effectiveness studies. We present the challenges encountered when constructing this dataset, beginning with an examination of data quality across data partners. We then discuss the methods and best practices used to operationalize several important study elements: exposure to treatment, baseline health comorbidities, and key outcomes of interest. RESULTS: We share our experiences and lessons learned when working with heterogeneous EHR data from over 65 healthcare institutions and 4 common data models. We discuss six key areas of data variability and quality. (1) The specific EHR data elements captured from a site can vary depending on source data model and practice. (2) Data missingness remains a significant issue. (3) Drug exposures can be recorded at different levels and may not contain route of administration or dosage information. (4) Reconstruction of continuous drug exposure intervals may not always be possible. (5) EHR discontinuity is a major concern for capturing history of prior treatment and comorbidities. Lastly, (6) access to EHR data alone limits the potential outcomes which can be used in studies. CONCLUSIONS: The creation of large scale centralized multi-site EHR databases such as N3C enables a wide range of research aimed at better understanding treatments and health impacts of many conditions including COVID-19. As with all observational research, it is important that research teams engage with appropriate domain experts to understand the data in order to define research questions that are both clinically important and feasible to address using these real world data.

Sample average treatment effect on the treated analysis using counterfactual explanation identifies BMT and SARS-CoV-2 vaccination as protective risk factors associated with COVID-19 severity and survival in patients with multiple myeloma (preprint)

Patients with multiple myeloma (MM), an age-dependent neoplasm of antibody-producing plasma cells, have compromised immune systems and might be at increased risk for severe COVID-19 outcomes. This study characterizes risk factors associated with clinical indicators of COVID-19 severity and all-cause mortality in myeloma patients utilizing NCATS' National COVID Cohort Collaborative (N3C) database. The N3C consortium is a large, centralized data resource representing the largest multi-center cohort of COVID-19 cases and controls nationwide (>16 million total patients, and >6 million confirmed COVID-19+ cases to date). Our cohort included myeloma patients (both inpatients and outpatients) within the N3C consortium who have been diagnosed with COVID-19 based on positive PCR or antigen tests or ICD-10-CM diagnosis code. The outcomes of interest include all-cause mortality (including discharge to hospice) during the index encounter and clinical indicators of severity (i.e., hospitalization/emergency department/ED visit, use of mechanical ventilation, or extracorporeal membrane oxygenation (ECMO)). Finally, causal inference analysis was performed using the propensity score matching (PSM) method. As of 05/16/2022, the N3C consortium included 1,061,748 cancer patients, out of which 26,064 were MM patients (8,588 were COVID-19 positive). The mean age at COVID-19 diagnosis was 65.89 years, 46.8% were females, and 20.2% were of black race. 4.47% of patients died within 30 days of COVID-19 hospitalization. Overall, the survival probability was 90.7% across the course of the study. Multivariate logistic regression analysis showed histories of pulmonary and renal disease, dexamethasone, proteasome inhibitor/PI, immunomodulatory/IMiD therapies, and severe Charlson Comorbidity Index/CCI were significantly associated with higher risks of severe COVID-19 outcomes. Protective associations were observed with blood-or-marrow transplant/BMT and COVID-19 vaccination. Further, multivariate cox proportional hazard analysis showed that high and moderate CCI levels, International Staging System (ISS) moderate or severe stage, and PI therapy were associated with worse survival, while BMT and COVID-19 vaccination were associated with lower risk of death. Finally, matched sample average treatment effect on the treated (SATT) confirmed the causal effect of BMT and vaccination status as top protective factors associated with COVID-19 risk among US patients suffering from multiple myeloma. To the best of our knowledge, this is the largest nationwide study on myeloma patients with COVID-19.

Risk and Outcome of Breakthrough COVID-19 Infections in Vaccinated Patients With Cancer: Real-World Evidence From the National COVID Cohort Collaborative.

PURPOSE: To provide real-world evidence on risks and outcomes of breakthrough COVID-19 infections in vaccinated patients with cancer using the largest national cohort of COVID-19 cases and controls. METHODS: We used the National COVID Cohort Collaborative (N3C) to identify breakthrough infections between December 1, 2020, and May 31, 2021. We included patients partially or fully vaccinated with mRNA COVID-19 vaccines with no prior SARS-CoV-2 infection record. Risks for breakthrough infection and severe outcomes were analyzed using logistic regression. RESULTS: A total of 6,860 breakthrough cases were identified within the N3C-vaccinated population, among whom 1,460 (21.3%) were patients with cancer. Solid tumors and hematologic malignancies had significantly higher risks for breakthrough infection (odds ratios [ORs] = 1.12, 95% CI, 1.01 to 1.23 and 4.64, 95% CI, 3.98 to 5.38) and severe outcomes (ORs = 1.33, 95% CI, 1.09 to 1.62 and 1.45, 95% CI, 1.08 to 1.95) compared with noncancer patients, adjusting for age, sex, race/ethnicity, smoking status, vaccine type, and vaccination date. Compared with solid tumors, hematologic malignancies were at increased risk for breakthrough infections (adjusted OR ranged from 2.07 for lymphoma to 7.25 for lymphoid leukemia). Breakthrough risk was reduced after the second vaccine dose for all cancers (OR = 0.04; 95% CI, 0.04 to 0.05), and for Moderna's mRNA-1273 compared with Pfizer's BNT162b2 vaccine (OR = 0.66; 95% CI, 0.62 to 0.70), particularly in patients with multiple myeloma (OR = 0.35; 95% CI, 0.15 to 0.72). Medications with major immunosuppressive effects and bone marrow transplantation were strongly associated with breakthrough risk among the vaccinated population. CONCLUSION: Real-world evidence shows that patients with cancer, especially hematologic malignancies, are at higher risk for developing breakthrough infections and severe outcomes. Patients with vaccination were at markedly decreased risk for breakthrough infections. Further work is needed to assess boosters and new SARS-CoV-2 variants.

A method for comparing multiple imputation techniques: a case study on the U.S. National COVID Cohort Collaborative (preprint)

Healthcare datasets obtained from Electronic Health Records have proven to be extremely useful to assess associations between patients' predictors and outcomes of interest. However, these datasets often suffer from missing values in a high proportion of cases and the simple removal of these cases may introduce severe bias. For these reasons, several multiple imputation algorithms have been proposed to attempt to recover the missing information. Each algorithm presents strengths and weaknesses, and there is currently no consensus on which multiple imputation algorithms works best in a given scenario. Furthermore, the selection of each algorithm parameters and data-related modelling choices are also both crucial and challenging. In this paper, we propose a novel framework to numerically evaluate strategies for handling missing data in the context of statistical analysis, with a particular focus on multiple imputation techniques. We demonstrate the feasibility of our approach on a large cohort of type-2 diabetes patients provided by the National COVID Cohort Collaborative (N3C) Enclave, where we explored the influence of various patient characteristics on outcomes related to COVID-19. Our analysis included classic multiple imputation techniques as well as simple complete-case Inverse Probability Weighted models. The experiments presented here show that our approach could effectively highlight the most valid and performant missing-data handling strategy for our case study. Moreover, our methodology allowed us to gain an understanding of the behavior of the different models and of how it changed as we modified their parameters. Our method is general and can be applied to different research fields and on datasets containing heterogeneous types.

Outcomes of COVID-19 in Patients With Cancer: Report From the National COVID Cohort Collaborative (N3C).

PURPOSE: Variation in risk of adverse clinical outcomes in patients with cancer and COVID-19 has been reported from relatively small cohorts. The NCATS' National COVID Cohort Collaborative (N3C) is a centralized data resource representing the largest multicenter cohort of COVID-19 cases and controls nationwide. We aimed to construct and characterize the cancer cohort within N3C and identify risk factors for all-cause mortality from COVID-19. METHODS: We used 4,382,085 patients from 50 US medical centers to construct a cohort of patients with cancer. We restricted analyses to adults ≥ 18 years old with a COVID-19-positive or COVID-19-negative diagnosis between January 1, 2020, and March 25, 2021. We followed N3C selection of an index encounter per patient for analyses. All analyses were performed in the N3C Data Enclave Palantir platform. RESULTS: A total of 398,579 adult patients with cancer were identified from the N3C cohort; 63,413 (15.9%) were COVID-19-positive. Most common represented cancers were skin (13.8%), breast (13.7%), prostate (10.6%), hematologic (10.5%), and GI cancers (10%). COVID-19 positivity was significantly associated with increased risk of all-cause mortality (hazard ratio, 1.20; 95% CI, 1.15 to 1.24). Among COVID-19-positive patients, age ≥ 65 years, male gender, Southern or Western US residence, an adjusted Charlson Comorbidity Index score ≥ 4, hematologic malignancy, multitumor sites, and recent cytotoxic therapy were associated with increased risk of all-cause mortality. Patients who received recent immunotherapies or targeted therapies did not have higher risk of overall mortality. CONCLUSION: Using N3C, we assembled the largest nationally representative cohort of patients with cancer and COVID-19 to date. We identified demographic and clinical factors associated with increased all-cause mortality in patients with cancer. Full characterization of the cohort will provide further insights into the effects of COVID-19 on cancer outcomes and the ability to continue specific cancer treatments.